Skip to content

feat(kindle-dedrm): concise manage and setup skills, measured no-loss - #12121

Merged
kyle-sexton merged 2 commits into
mainfrom
feat/6609-concise-kindle-dedrm
Nov 1, 2026
Merged

feat(kindle-dedrm): concise manage and setup skills, measured no-loss#12121
kyle-sexton merged 2 commits into
mainfrom
feat/6609-concise-kindle-dedrm

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 1, 2026

Copy link
Copy Markdown
Member

Closes #6609

Summary

Concise rewrite of the kindle-dedrm plugin's two skills, manage and setup, measured before and after on Opus 5.5 and Sonnet 5.5 under the rewrite protocol. Every measured part showed no loss on both models, so all of them ship.

Part Decided on Opus 5.5 before -> after Sonnet 5.5 before -> after Result
manage description (888 -> 621 chars) 16 held-out trigger probes x 3 runs fired 21/24 -> 21/24, quiet 24/24 -> 24/24 fired 21/24 -> 21/24, quiet 24/24 -> 24/24 ships
manage body (2009 -> 1907 words) md-* outcome cases, 5 runs each mean 0.933 -> 0.908 (4 cases; md-installer-popup unmeasured) mean 0.933 -> 0.933 (5 cases) ships
setup body (917 -> 853 words) su-check-wrong-version, 5 runs 1.000 -> 1.000 1.000 -> 1.000 ships
setup description none (disable-model-invocation: true) n/a n/a unchanged

Per case (with-arm mean, before -> after):

Case Opus 5.5 Sonnet 5.5
md-share-keys-refused 1.000 -> 1.000 1.000 -> 1.000
md-friend-files 0.733 -> 0.667 0.667 -> 0.667
md-cleanup-batch-library 1.000 -> 0.967 (rerun) 1.000 -> 1.000
md-installer-popup unmeasured (harness denial on both versions) 1.000 -> 1.000
md-sync-routes 1.000 -> 1.000 (rerun) 1.000 -> 1.000
su-check-wrong-version 1.000 -> 1.000 1.000 -> 1.000

Fix

  • manage description: drops the per-action step lists and the sentence "Every state mutation has a documented compensating reversal". It keeps both quoted trigger phrases, every action name, "Windows only", the owned-books scope and every other trigger clause.
  • manage body: drops the "uniform check/apply contract" provenance asides, the "single owner" restatement, and the duplicate no-mutation sentences in update, and tightens the Purpose and auto-detect wording. The Hard safety rules, the sync sequence, the reversal matrix, the Calibre Library rule, "What this skill does NOT do", every code block and the pointer record are byte-identical.
  • setup body: drops the marketplace setup-contract pointer, the no-write clause that "What this skill does NOT do" already states, a duplicated apply scope sentence, and framing words. The check items, the apply commands and the safety rules are unchanged.
  • Changelog fragment .changes/kindle-dedrm/ (patch).

A fresh agent wrote the rewrite. It could not read the eval cases, the probes or this issue, and it was limited to five classes of cut: citation duplicates, citations of citations, provenance asides, non-safety duplicates and flavor.

Verification

  • Frozen suite. plugins/kindle-dedrm/evals git tree d88068509e, unchanged from fix(kindle-dedrm): call sync-finalize.sh without its nonexistent argument, keep cleanup per-item and the library kept, and add manage and setup eval cases #11967. The same case files ran on both versions, with one run-copy edit that was never committed: the md-sync-routes Skill routing grader was set to with-only. Under --ablation none only the with-arm runs either way.
  • Held-out trigger set. 16 queries, 8 should-trigger and 8 should-not. A fresh agent wrote them blind from a neutral purpose statement, and they were kept outside the repository. Neither the rewriter nor the orchestrating session read them query by query.
  • Run conditions. Claude Code 2.1.316. Opus 5.5 and Sonnet 5.5 at --runs 5 for cases and --runs 3 for probes. --ablation none, judge opus (no llm grader). Before and after were launched together for every pair. A sandbox probe ran first and confirmed that no Windows drive, Windows binary or case path was reachable before Bash was granted.
  • Pre-registered rules. Written before any run, see the decision rule.
    • Each part ships only if every read pass is VALID under run-validity.py.
    • The mean case score holds within 0.05.
    • No case drops by a third or more.
    • No must-not grader fails more often after than before.
    • The safety-edge cases (md-share-keys-refused, md-friend-files, md-cleanup-batch-library) are measured on both models.
    • For the description: no lost trigger, and fired/quiet counts within one run.
  • Meaning diff. Two fresh docs-hygiene:compare-labeler agents labeled each skill, and all four returned PASS: no semantic loss, no ambiguity.
  • Denials. On Opus, three passes went INVALID on a harness denial class: the agent globbed or sed-read the plugin copy's own directory, about one run in five, on either version. Each got its one pre-registered paired rerun. Two reruns came back VALID. md-installer-popup hit the same class on both versions again, so it is unmeasured on Opus and dropped from both sides. Its graders scored 1.000 on every run of both versions. Sonnet had no denials.
  • Misses, read in full.
    • md-friend-files my-kindle-content misses on both versions and both models are real. The replies decline but never name the folder, and the count is equal or better before.
    • One Opus declines-not-owned miss and two per-item misses (one before, one after) meet the expected outcome. They are grader recall gaps, listed under Related.
  • /skill-quality:check: PASS before and after, with the same warning set and smaller sizes.
  • gitleaks over the branch: no leaks.
  • Cost. 35 passes, about 30.3 USD (list-price estimate from the CLI's costUsd).

The frozen case descriptions cite SKILL.md line numbers of the before text. Those citations still refer to it.

Related

🤖 Generated with Claude Code

kyle-sexton and others added 2 commits November 1, 2026 16:18
@kyle-sexton @claude
feat(kindle-dedrm): shorten the manage description and trim the manag…
2bfd70e 
…e and setup bodies

Ships the #6609 concise rewrite, measured with no loss on Opus 5.5 and
Sonnet 5.5. The manage description drops the per-action step lists and
keeps both quoted trigger phrases; held-out trigger probes fired and
stayed quiet the same on both versions. The manage and setup bodies drop
provenance asides, duplicates and framing; the safety rules, steps and
commands are unchanged, and the outcome cases showed no loss.

Closes #6609

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
Merge remote-tracking branch 'origin/main' into feat/6609-concise-kin…
0ba42b3 
…dle-dedrm
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 1, 2026 21:41
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 1, 2026 •
edited
Loading

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-01T21:43:04.567441Z 0ba42b3 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 0ba42b3e2080197b20be8b5552a6cf372b84f56b: read 3 of 3 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #12121 (kindle-dedrm concise skills)

No security issues found.

I read every hunk in all 3 changed files: the changelog fragment, plugins/kindle-dedrm/skills/manage/SKILL.md and plugins/kindle-dedrm/skills/setup/SKILL.md. None of them matches a trusted-path glob. No plugin manifest changed, so no manifest <field> lines apply.

The diff shortens standing skill instructions, so I checked each removed sentence under the instruction-surface lens. I judged each one by its consequence: what would break if the rule were gone. docs/contracts/instruction-exception-register/README.md was not consulted, so the protected-class list was not checked.

  • manage description: drops the per-action step lists and the sentence "Every state mutation has a documented compensating reversal." That sentence describes the skill. It is not a rule. The reversal matrix and the per-item confirmation rules for cleanup are untouched, and the description still says "per-item confirmation."
  • manage update / re-pin: "gated on a working-tree checkout" becomes "apply it only when ${CLAUDE_PLUGIN_ROOT} is inside a git working tree". The rev-parse check, the stop for an installed (cache) copy and "never edit bundled files in the cache" are all still there. What the rule forbids has not changed.
  • manage auto-detect: "Never commit to setup/sync/cleanup without user confirmation when ambiguous" survives with the same meaning. The Hard safety rules section is outside every hunk, so it is unchanged.
  • setup Purpose: drops the clause "Setup writes no Claude Code user settings, no pluginConfigs, and nothing into the plugin cache or plugin data directory." The same rule is still in force at setup/SKILL.md#L94, under "What this skill does NOT do." The pause at every CHECKPOINT, the check rule to modify, download and extract nothing, and the pointer to workflow.md as the only source of the steps are unchanged.

The PR description and issue comments contain no embedded instructions aimed at this review.

Spend: the PR reports about 30.3 USD of prepaid eval runs that were already done. This change triggers no billing.

critical-findings: 0

@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

claude-review reviewed 0ba42b3e2080197b20be8b5552a6cf372b84f56b: read 3 of 3 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #12121 (kindle-dedrm concise manage and setup skills)

No findings. I read every hunk of all 3 changed files.

What changed. The PR only edits Markdown text. It adds a changelog fragment and shortens the wording of plugins/kindle-dedrm/skills/manage/SKILL.md and plugins/kindle-dedrm/skills/setup/SKILL.md. I checked each removed sentence against what is still in the head files:

  • manage description. All five action names, "Windows only", the owned-books scope, both quoted trigger phrases and the other trigger clauses are still there. The cut sentence about compensating reversals is still covered by the body's reversal matrix and Purpose section.
  • manage body. The update section still says "No mutations" once, and still requires a working-tree checkout before a re-pin edit. The ambiguity rule is kept in the auto-detect paragraph ("When ambiguous, never commit to setup/sync/cleanup without user confirmation"). The diff touches no code block, safety rule or reversal-matrix row.
  • setup body. The removed sentence saying setup writes no Claude Code settings, pluginConfigs, plugin cache or plugin data is still stated under "What this skill does NOT do" (setup/SKILL.md#L94). The apply download scope removed from the apply intro is still in Purpose and in the apply download bullet. "The status script below" correctly points at the status.sh block that follows it.

Path rules. I checked .claude/rules/skill-bodies-state-current-rules.md and .claude/rules/cost-claims.md, which both match plugins/*/skills/**. The edits add no restated upstream detail and no price or cost figure. They only remove text.

Closing issue #6609. The "Done when" items are met in the PR body:

  • The PR body has a per-skill decision table.
  • The parts left unmeasured, and why, are named: the setup description is unchanged because of disable-model-invocation: true, and md-installer-popup was unmeasured on Opus.

The measurements themselves (scores, probe counts, the ~30.3 USD run cost) are the author's claims. I did not run or check them.

Changelog fragment. bump: patch fits a text-only rewrite that changes no behaviour. No plugin manifest changed, so the manifest checks don't apply.

Embedded text. The PR context and prior-findings files contain no instructions aimed at this lane.

Prior findings. None (round 0 of 3).

Spend. None. The change starts no charge and adds no run spend.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 1, 2026
Merged via the queue into main with commit 30b8fbf Nov 1, 2026
67 checks passed
@kyle-sexton
kyle-sexton deleted the feat/6609-concise-kindle-dedrm branch November 1, 2026 21:56
This was referenced Nov 2, 2026
Merged
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Concise skills rollout: kindle-dedrm (2 skills, 1 model-invoked)

1 participant

@kyle-sexton